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Final  Report 

Person  Identification  from  Video  with  Multiple  Biometric  Cues: 
Benchmarks  for  Human  and  Machine  Performance 
PI :  Alice  J.  O’Toole,  The  University  of  Texas  at  Dallas 


Objectives 

The  objectives  of  the  proposed  work  are  divided  into  two  parts.  First,  we  have  developed  a 
database  of  person  images  and  videos  designed  to  provide  comprehensive  and  controlled 
tests  of  the  automatic  systems  developed  in  the  human  identification  at  a  distance  DARPA 
initiative.  Second,  we  have  made  use  of  this  database  to  assess  human  identification 
performance  under  a  variety  of  training  and  testing  conditions.  This  assessment  has  served 
as  a  baseline  for  evaluating  the  performance  of  computational  algorithms  in  the  DARPA 
initiative  under  the  Face  Recognition  Vendor  Test  conducted  this  year  at  the  National 
Institutes  of  Standards  and  Technology. 

Approach  &  Methods 

Part  1.  We  compiled  a  database  consisting  of  a  series  of  close-range,  surveillance-range, 
and  gait  videos  of  a  set  of  male  and  female  individuals  of  several  races.  The  close-range 
videos  capture  dynamic  information  about  the  face  across  a  range  of  emotional  expressions. 
A  subset  of  these  videos  include  speech-generated  lip-movements.  The  surveillance-range 
videos  include  dynamic  information  about  individuals  who  are  either  actively  or  passively 
involved  in  a  conversation.  Gait  videos  are  taken  of  subjects  walking  and  include  information 
about  the  face,  posture,  and  gait  of  the  individual. 

Part  2.  Three  series  of  experiments  have  been  performed.  The  first  series  of  experiments 
provides  baseline  data  for  human  recognition  with  different  presentation  types.  These 
experiments  test  recognition  when  learning  and  testing  presentation  types  match.  The  second 
series  of  experiments  provide  data  for  human  recognition  when  learning  and  testing  data 
presentation  types  do  not  match.  We  have  tested  recognition  transfer  conditions  that  are 
encountered  commonly  and  unavoidably  in  the  situations  targeted  in  this  DARPA  initiative. 
The  third  series  of  experiments  addresses  questions  about  the  nature  of  familiarity  in  person 
recognition.  Previous  psychophysical  results  indicate  that  people  are  far  more  accurate  at 
generalizing  recognition  across  presentation  conditions  when  they  are  familiar  with  the 
person  to  be  identified  than  when  the  person  is  unfamiliar  to  them.  Using  the  various 
categories  of  close  range  and  gait  videos,  we  have  varied  the  learning  and  testing  conditions 
to  determine  which  combinations  of  learning  stimuli  facilitate  familiarity,  and  thus,  accurate 
generalization  of  identification  to  novel  presentation  types. 

Status 

We  have  compiled  a  database  of  images  and  videos  that  includes  images  and  videos  of 
approximately  300  human  participants.  Duplicate  sets  of  images  and  videos  taken  from  1  week 
to  6  months  after  the  first  set  are  available  for  approximately  200  of  these  people.  The  images 
include  9  standard  “mug  shot  poses”.  The  videos  include  head  rotations,  dynamic  facial 
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expressions,  facial  speech  clips,  and  3  whole-body  gait  clips.  Experiments  have  been  completed 
comparing  the  effects  of  several  types  of  facial  motion  on  face  recognition,  the  effects  of  face 
familiarity  on  recognition  from  video  clips  taken  at  a  distance,  and  on  the  effects  of  attention  on 
recognition  of  moving  faces.  The  results  of  these  studies  provide  insight  into  the  way  motion  can 
facilitate  or  interfere  with  the  encoding  of  the  invariant  face  and  body  features  that  support 
recognition. 

Accomplishments,  Publications,  and  Presentations 

The  accomplishments  of  the  project  are  reported  below  in  the  form  of  the  DARPA  quarterly 
reports  that  span  the  final  year  of  this  grant.  In  these  reports,  we  list  publications  and  conference 
presentations,  as  required  for  the  Publication  (section  6)  and  Interaction  (section  7)  sections  of 
the  AFOSR  final  report. 


Quarterly  Report  -  January  15, 2002 
Human  Identification  at  a  Distance  Project 

Progress  Summary1 

■  The  first  two  experiments  on  the  effects  of  facial  movement  on  recognition  have  produced 
preliminary  results  (see  submitted  abstract  reference  below).  We  continue  to  collect 
additional  data  on  these  experiments  in  order  to  increase  the  power  of  the  designs. 

■  Snow,  S.  M.,  Lannen,  G.  M.,  O’Toole,  A.  J.  &  Abdi,  H.  Memory  for  moving  faces: 
Effects  of  rigid  and  non-rigid  facial  motion,  (abstract  submitted  for  presentation  to  The 
Annual  Vision  Science  Society  Meeting). 

■  The  second  set  of  research  involved  a  review  and  synthesis  of  the  psychological  literature  on 
recognition  of  moving  faces  (see  recently  submitted  abstract  reference  and  accepted  paper 
reference). 

■  O’Toole,  A.  J.,  Roark,  D.  Abdi,  H.  Recognizing  moving  faces:  A  Psychological  and 
Neural  Framework,  (abstract  submitted  for  presentation  to  The  Annual  Vision  Science 
Society  Meeting). 

■  O’Toole,  A.  J.,  Roark,  D.  Abdi,  H.  Recognizing  moving  faces:  A  Psychological  and 
Neural  Framework  (manuscript  accepted  with  revisions)  Trends  in  Cognitive  Sciences. 

m 

■  Two  additional  experimental  projects  are  in  the  planning  and  preparation  stage.  An 
investigation  of  the  interaction  between  color  and  motion  information  in  recognition  memory 
is  ready  for  data  collection.  A  second  experiment  examining  the  role  of  familiarity  in  the  use 
of  motion  information  for  identification  is  nearly  ready  for  data  collection. 

f  The  database  was  continually  expanded  so  that  it  now  includes  250  data  collections 
comprised  of  148  persons  with  duplicate  sessions  on  102  of  these. 

Recent  Progress 

From  October  15,  2001  to  January  15,  2002,  we  have  accomplished  the  following: 

Work  Performed. 
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Subject  filming.  We  have  added  to  the  database  on  a  continuous  basis  over  this  time.  We 
now  have  250  full  sets  of  the  protocol  still  and  video  shots.  This  includes  148  people 
with  duplicate  sessions  on  102  people.  Editing  of  these  videos  proceeded  in  parallel 
with  the  data  collection  effort  and  is,  at  present,  nearly  up  to  date. 

Database  development.  We  have  been  working  on  the  problem  of  converting  our  UTD- 
developed  MS  Access  database  to  the  recently  finalized  DARPA  XML  database.. 

Psychophysical  Experiments.  We  now  have  preliminary  data  on  two  experiments. 
Abstracted  versions  of  these  have  been  forwarded  previously  to  the  PM.  The  abstract 
from  one  of  these  has  been  submitted  formally  to  a  conference  for  presentation,  (The 
2nd  Annual  Meeting  of  the  Vision  Science  Society).  A  paper  in  which  we  review  the 
psychological  data  on  human  memory  for  moving  faces  has  been  prepared,  submitted, 
and  accepted  with  minor  revisions  to  the  journal  Trends  in  Cognitive  Sciences, 
O’Toole,  A.  J.  Roark,  D.  &  Abdi,  H.  (accepted  with  revisions).  Recognition  of 
moving  faces:  A  Psychological  and  Neural  Framework.  We  are  currently  beginning 
data  collection  for  three  additional  experiments  aimed  at  assessing  the  effects  of 
different  kinds  of  movement  on  recognition  memory  for  faces. 

Progress  Summary 

■  The  database  now  contains  148  persons  with  duplicate  sessions  on  102  people.  We  will 
proceed,  this  quarter,  to  collect  additional  videos  in  a  systematic  fashion  and  edit/log  these  as 
we  go. 

■  The  first  two  experiments  have  produced  preliminary  results  (see  submitted  abstract 
references  below).  We  continue  to  collect  additional  data  on  these  experiments  in  order  to 
increase  the  power  of  the  designs  enough  to  test  for  statistical  significance. 

■  Three  additional  experimental  projects  are  planned.  An  investigation  of  the  interaction 
between  color  and  motion  information  in  recognition  memory  is  ready  for  data  collection.  A 
second  experiment  examining  the  role  of  familiarity  in  the  use  of  motion  infonnation  for 
identification  is  nearly  ready  for  data  collection.  An  investigation  of  the  role  of  expression 
perception  on  memory  for  the  identity  of  the  person  is  also  planned. 

■  O’Toole,  A.  J.,  Roark,  D.  Abdi,  H.  Recognizing  moving  faces:  A  Psychological  and  Neural 
Framework,  (abstract  submitted  for  presentation  to  The  Annual  Vision  Science  Society 
Meeting). 

■  Snow,  S.  M.,  Lannen,  G.  M.,  O’Toole,  A,  J.  &  Abdi,  H.  Memory  for  moving  faces:  Effects  of 
rigid  and  non-rigid  facial  motion,  (abstract  submitted  for  presentation  to  The  Annual  Vision 
Science  Society  Meeting). 

■  O’Toole,  A.  J.,  Roark,  D.  Abdi,  H.  Recognizing  moving  faces:  A  Psychological  and  Neural 
Framework  (manuscript  accepted  with  revisions)  Trends  in  Cognitive  Sciences.  Monthly 
Report  -  March  15,  2002 
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Recent  Progress 

From  January  15, 2001  to  April  15, 2002,  we  have  accomplished  the  following: 

Work  Performed. 

Database  development.  We  have  completed  six  of  the  eight  required  files  for  the  XML 
coded  database  and  are  working  on  the  remaining  two,  which  will  involve  more 
complex  conversions  of  the  data.  We  hope  to  produce  tools  that  will  automatize  this 
been  working  on  the  problem  of  converting  our  UTD-developed  MS  Access  database  to 
the  recently  finalized  DARPA  XML  database. 

Psychophysical  Experiments.  Data  collection  has  begun  on  five  new  experiments.  Two  of 
these  are  extensions  of  work  described  previously.  The  first  involves  the  effects  of 
head  motion  on  recognition  memory  for  faces.  A  second  extension  of  our  previous 
findings  is  to  compare  the  effects  of  four  kinds  of  motion  on  recognition  memory 
accuracy.  We  are  comparing  the  effects  of  head  rotations,  facial  speech,  and 
expression  in  a  single  experimental  paradigm.  Additional  experiments  have  begun  on: 
1.)  the  effects  of  familiarization  of  moving  faces  on  recognition  accuracy;  2.)  the 
effects  of  facial  expression  and  motion  on  recognition  accuracy;  and  the  3.)  combined 
effects  of  color  and  motion  on  recognition.  We  anticipate  the  completion  of  data 
collection  on  these  experiments  by  the  end  of  May,  2002. 

■  Subject  filming.  We  have  added  to  the  database  on  a  continuous  basis  over  this  time. 


Progress  Summary 

■  All  five  experiments  described  above  are  in  progress  and  will  be  complete  by  May,  2002.. 

■  XML  code  for  six  of  eight  required  database  files  is  complete  and  will  be  forwarded  to  the 
database  managers  within  1  week  of  this  writing.  Conversion  code  for  the  7th  file  is  nearly 
complete. 

■  We  now  have  161  subjects  in  the  database,  including  full  sets  of  static  and  video  images  for 
each.  Duplicate  sessions  exist  for  106  of  these  individuals,  triplicates  exist  for  3  individuals 
and  quadruplicates  for  1  individual. 
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Recent  Progress 

From  April  16, 2001  to  July  15, 2002,  we  have  accomplished  the  following: 

Work  Performed. 

Database  development.  We  have  completed  six  of  the  eight  required  files  for  the  XML 
coded  database  and  are  working  on  the  remaining  two,  which  will  involve  more 
complex  conversions  of  the  data. 

Psychophysical  Experiments.  Data  collection  has  been  completed  on  the  following 
experiments.  We  have  completed  a  comparison  of  head  movements  including  faces; 
speech,  expression  and  head  rotation.  We  have  extended  the  previous  work  on  head 
rotation  to  test  the  transfer  to  a  single  static  image  and  are  currently  in  the  process  of 
writing  a  paper  on  the  findings.  Several  additional  experiments  are  underway  to:  a.) 
assess  the  effects  of  attention  load  on  face  recognition,  by  varying  the  predictability  of 
facial  expressions  on  a  trial  by  trial  basis;  b.)  assess  the  effects  of  color  and  motion  on 
recognition  of  static  and  moving  stimuli;  c.)  further  compare  the  effects  of  head 
movements  when  transferring  to  single  frontal  images;  and  d.)  assess  the  effects  of 
familiarity  on  learning  moving  versus  static  presentations  of  faces.  We  anticipate  the 
completion  of  data  collection  on  these  experiments  by  the  end  of  August,  2002. 

■  Subject  filming.  We  have  added  to  the  database  on  a  continuous  basis  over  this  time. 


Progress  Summary 

■  The  first  two  experiments  described  have  been  completed  and  paper  preparation  is  in 
progress. 

■  XML  code  for  six  of  eight  required  database  files  is  complete  and  have  been  forwarded  to  the 
database  managers. 

*  We  now  have  210  subjects  in  the  database,  including  full  sets  of  static  and  video  images  for 
each.  Duplicate  sessions  exist  for  143  of  these  individuals,  triplicates  exist  for  9  individuals 
and  quadruplicates  for  3  individual. 

■  We  have  just  published  a  review  of  the  literature  on  memory  for  moving  faces. 

■  O’Toole,  A.  J.,  Roark,  D.,  and  Abdi,  H.  Trends  in  Cognitive  Sciences,  6, 261-266. 
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Quarterly  Report  -  October  15, 2002 
Human  Identification  at  a  Distance  Project 

Recent  Progress 

From  July  16,  2002  to  October  15,  2002,  we  have  accomplished  the  following: 

Work  Performed, 

Database  development.  XML  coding  of  the  database  is  nearly  complete.  The  recent  changes 
in  the  dtd  files  from  the  Hbase  developers  delayed  the  final  conversion. 

Psychophysical  Experiments.  Data  collection  has  been  completed  on  the  following 
experiments.  We  have  completed  a  comparison  of  head  movements  including  facial 
speech,  expression,  and  head  rotation.  We  have  extended  the  previous  work  on  head 
rotation  to  test  the  transfer  to  a  single  static  inage.  A  final  follow-up  experiment  is 
underway.  Data  collection  is  nearly  complete  in  several  additional  experiments  to:  a.) 
assess  the  effects  of  attention  load  on  face  recognition,  by  varying  the  predictability  of 
facial  expressions  on  a  trial  by  trial  basis;  b.)  further  compare  the  effects  of  head 
movements  when  transferring  to  single  frontal  images;  and  c.)  assess  the  effects  of 
familiarity  on  learning  moving  versus  static  presentations  of  faces.  This  third 
experiment  has  produced  interesting  findings  indicating  that  familiarity  with  the  face 
may  be  necessary  for  motion  to  have  a  beneficial  effect  on  memory.  We  are  also 
currently  looking  at  the  effects  of  color  and  motion  on  recognition  of  static  and 
moving  stimuli.  We  anticipate  the  completion  of  data  collection  on  these  experiments 
by  the  end  of  November  2002. 

■  Subject  filming.  We  have  added  to  the  database  on  a  continuous  basis  over  this  time, 
although  we  have  slowed  down  this  collection  temporarily  to  be  able  to  proceed  with  the 
testing  of  subjects  in  memory  experiments. 

Progress  Summary 

■  Paper  preparation  has  begun  for  all  of  the  experiments  in  which  data  collection  is  complete.. 

■  XML  code  conversion  of  the  database  files  is  nearly  complete. 

■  We  now  have  250  subjects  in  the  database,  including  full  sets  of  static  and  video  images  for 
each.  Duplicate  sessions  exist  for  180  of  these  individuals,  triplicates  exist  for  9  individuals 
and  quadruplicates  for  3  individual. 

*  We  have  just  submitted  an  extended  review  paper  of  the  literature  on  memory  for 
moving  faces  to  the  journal  Behavior  and  Cognitive  Neuroscience  Reviews.  This  is  a 
much  more  highly  detailed  version  of  the  O’Toole,  Roark,  and  Abdi,  (2002).  Trends  in 
Cognitive  Sciences,  6,  261-266.  The  paper  also  includes  a  section  on  the  development  of 
recognition  of  moving  faces. 


Final  Report 
7 


Quarterly  Report  -  January  15, 2003 
Human  Identification  at  a  Distance  Project 


Recent  Progress 

From  October  16, 2002  to  January  15, 2003,  we  have  accomplished  the  following: 

Work  Performed. 

Database  development,  XML  coding  of  the  database  is  complete  and  has  been  accepted  bby 
the  Hbase  developers. 

Psychophysical  Experiments.  Data  collection  has  continued  on  the  following 
experiments.  1)  The  experiment  assessing  the  effects  of  attention  load  on  face 
recognition,  by  varying  the  predictability  of  facial  expressions  on  a  trial  by  trial  basis 
is  about  2/3  complete.  We  have  included  full  counterbalances  on  the  gender  and  race 
of  the  subjects,  both  of  which  appear  to  be  affecting  performance.  2.)  The  experiment 
assessing  the  effects  of  familiarity  on  learning  moving  versus  static  presentations  of 
faces  is  nearly  complete.  Early  indications  are  that  motion  will  aid  in  recognition  and 
that  there  may  be  an  interaction  between  familiarity  and  motion  such  that  motion 
becomes  relatively  more  important  as  the  faces  become  more  familiar.  3.)  We  are  also 
currently  looking  at  the  effects  of  color  and  motion  on  recognition  of  static  and 
moving  stimuli.  4.)  An  experiment  using  synthetic  versions  of  faces  is  being  carried 
out  to  look  at  short  term  perceptual  high  level  adaptation  effects.  We  anticipate  the 
completion  of  data  collection  on  these  experiments  by  the  end  of  February,  2003. 

*  Subject  filming.  We  have  added  to  the  database  on  a  continuous  basis  over  this  time, 
although  we  have  slowed  down  this  collection  temporarily  to  be  able  to  proceed  with  the 
testing  of  subjects  in  memory  experiments. 

Progress  Summary 

■  Paper  preparation  has  begun  for  all  of  the  experiments  in  which  data  collection  is  complete.. 

■  XML  code  conversion  of  the  database  files  is  complete  and  submitted. 

■  We  now  have  over  300  subjects  in  the  database,  including  full  sets  of  static  and  video  images 
for  each.  Duplicate  sessions  exist  for  over  200  of  these  individuals,  triplicates  exist  for  9 
individuals  and  quadruplicates  for  3  individual. 

■  We  are  preparing  a  manuscript  for  the  IEEE  Transactions  on  Pattern  Analysis  and  Machine 
Vision  on  the  database,  which  will  be  made  publicly  available  when  this  paper  is  published. 

■  An  extended  review  paper  of  the  literature  on  memory  for  moving  faces  has  been 
accepted  to  the  journal  Behavior  and  Cognitive  Neuroscience  Reviews.  This  is  a  much 
more  highly  detailed  version  of  the  O’Toole,  Roark,  and  Abdi,  (2002).  Trends  in 
Cognitive  Sciences,  6,  261-266.  The  paper  also  includes  a  section  on  the  development  of 
recognition  of  moving  faces.  We  anticipate  publication  early  this  year. 
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